AI localization

The AI localization engine uses a neural network to find objects in the scan. It needs the AI Localization software module and is used by the AI Localization (LS) and AI Picking (BPS) solution types. See Software modules.

This page describes the settings of the AI localization engine:

• General settings – the neural network, the object score, the empty scene check, the bounding box and how the gripping point is placed.

• Gripping point settings – the fine-tuning of the gripping point for each placement method.

• Dimensions reporting – the dimensions and the angle that the engine reports for each object.

Where to find the settings

The settings are stored in the localization profile of the vision system (a .pacf file). To change them, open the Localization tab of the vision system and click Edit current (or Create new for a new profile). This opens the localization profile editor. The buttons work the same as for CAD localization, see Localization profile editor.

The localization profile editor of the AI Picking [Tote] example solution after Scan & Locate: four totes found. Red frames: the editor tabs, Scan & Locate, Locate, Save configuration and Quit localization.

Compared with the CAD editor:

• There are only the Localization and Texture tabs (no model features) and no Localization model source.

• There are no stop criteria.

• The Localized objects table shows Score, Duration [ms], Label and Dimension X/Y [mm] for each object.

The settings are below the visualizer, in three groups: General, Gripping point - Basic and Gripping point - Advanced (GP refinement). Which gripping point settings are shown depends on the Initial gripping point placement method.

The localization settings of the AI Picking [Tote] example solution, with the Center of Tote method. Reset to defaults is framed in red. Highlighted values differ from the defaults.

General settings

Initial gripping point placement method

How the engine places the gripping point on a found object. Choose the method that suits the objects in your application. There are two groups of methods:

• Dedicated methods are made for one object type. Each has its own set of gripping point settings.

  • Center of Box

  • Center of Tote

• General methods work for other objects. They share one set of gripping point settings.

  • Centroid

  • Largest Inscribed Circle

  • Keypoint from NN

Each method is described below.

Center of Box

For flat objects, such as the side of a box.

• Position: the middle of the top face of the object.

• Orientation: the X-axis is parallel with the longer side of the box. Its direction:

  • 3D bounding box enabled: the positive X-axis points the same way as the X-axis of the 3D bounding box (the angle between them is less than 90°).

  • 3D bounding box disabled: the direction of the X-axis is ambiguous.

Center of Tote

For totes (KLTs). The engine uses the top face of the tote (the rim). The calculated tote dimensions depend on the Tote edge detection point set setting (inner or outer points).

• Position: the middle of the plane of the tote rim.

• Orientation: depends on whether the neural network was trained with keypoints. A keypoint is a special feature on the tote or its contents that shows the correct orientation of the tote.

  • Neural network with keypoints: two diagonal lines through the gripping point (GP) divide the tote into four quadrants: left, right, top and bottom. The positive X-axis points into the quadrant that contains the keypoint. The X- and Y-axes are always parallel with the tote rims.

    One keypoint (KP_0): +X points into the quadrant that contains KP_0.

    

    The tote seen from the camera. Left: KP_0 is in the left quadrant (shaded), so +X points left. Right: KP_0 is in the top quadrant, so +X points up.

    Two keypoints (KP_0 and KP_1): the engine takes the arrow from KP_1 to KP_0 and moves it so that it starts at the GP. The moved KP_1 is called KP_11 (it lies on the GP) and the moved KP_0 is called KP_00. +X points into the quadrant that contains KP_00.

    

    Left: the neural network finds KP_1 and KP_0. Right: the arrow from KP_1 to KP_0 is moved to the GP. Its end, KP_00, is in the right quadrant (shaded), so +X points right.

  • Neural network without keypoints: the positive X-axis points the same way as the X-axis of the reference coordinate system (camera space). The angle between them is less than 90°.

Centroid

The gripping point is placed at the centroid of the object (segment).

Largest Inscribed Circle

The gripping point is placed at the center of the largest circle that fits inside the object (segment).

Keypoint from NN

The gripping point is placed at the keypoint of the object (segment) that the neural network was trained to find. A keypoint is a special feature on the object.

Use this method only with a neural network that provides a keypoint position. This depends on how the network was trained. If the network has no keypoint, the engine uses Centroid instead and divides the Object score of every found object by 10 (for example, 0.9 becomes 0.09). Objects are then found only if you lower Minimal object score.

For Centroid, Largest Inscribed Circle and Keypoint from NN, the orientation is the same: the X-axis is parallel with the longer side of the bounding box of the object. Its direction:

• 3D bounding box enabled: the positive X-axis points the same way as the X-axis of the 3D bounding box (the angle between them is less than 90°).

• 3D bounding box disabled: the direction of the X-axis is ambiguous.

The other general settings:

Neural Network

The neural network used to recognize the objects.

Minimal object score

The lowest score a candidate found by the neural network must have. Candidates with a lower score are rejected.

Use the Empty scene feature

When enabled, the localization stops early and returns Empty scene if the number of points in the scanning volume (or in a bounding box) is below the Empty scene threshold.

Empty scene threshold

The highest number of points in the scanning volume (or in a bounding box) at which the scene counts as empty. Below this number the localization stops early and returns Empty scene.

Crop AI input according to the bounding box

When enabled, and the 3D bounding box is enabled, the texture used for AI recognition is cropped to the projection of the 3D bounding box.

Reject objects crossing bounding box

When enabled (the default), objects (segments) outside the 3D bounding box and objects only partly inside it are rejected. When disabled, objects partly inside the 3D bounding box are accepted if their gripping point lies inside the box.

Note

The gripping point is computed on the whole segment reported by the neural network. A segment can be cut off by the 2D bounding box that is applied to the texture before AI recognition. This 2D crop is turned on and off with Crop AI input according to the bounding box.

Gripping point settings

Which gripping point settings are available depends on the Initial gripping point placement method. The settings are in two groups:

• Basic – the settings you usually adjust to get the result you want.

• Advanced (GP refinement) – settings for special or difficult cases, to fine-tune the result.

Each setting below is marked (Basic) or (Advanced).

Dedicated methods

Center of Box

Box plane inliers distance tolerance (Basic)

The largest distance a 3D point can be from the detected box plane and still count as part of it. Lower values make the plane detection stricter. Higher values are more tolerant, but also accept more outliers.

Minimum plane inliers threshold (Advanced)

The lowest percentage of points of the segmented box that must lie on the detected plane for the plane to be valid. This makes sure the plane really represents the box surface.

Expected plane inliers threshold (Advanced)

The percentage of points of the segmented box that you expect to lie on the detected plane. This estimate helps the plane search run faster.

Plane sampled points count (Advanced)

The number of points picked at random from the segmented box in each step of the plane search, to estimate a plane on top of the box. More points can give a more accurate result, but take longer to compute.

Center of Tote

Tote edge detection point set (Basic)

Which points count as the edge points of the tote:

• Inner Points – points on the inner edge of the tote.

• Outer Points – points on the outer edge of the tote.

This setting also changes the reported dimensions of the tote.

Tote edge fitting tolerance (Basic)

The distance (in millimeters) within which points are grouped together when lines are fitted to the tote edges. Points closer to each other than this belong to the same line segment. A smaller value gives a more precise but possibly less stable fit. A larger value is more forgiving but can lose detail.

Line fit center weight radius (Basic)

How much weight points get based on their distance from the center. Values between 0 and 1 give more weight to points near the center and less to points near the edges. With 0 or 1, all points have the same weight.

Segment border narrow (Advanced)

A narrow border width for precise detection of the tote rim plane. It draws a thinner outline around the detected tote, which gives more detailed edge detection. Best for clean, well-defined tote edges.

Tote edge fitting (Advanced)

A wider border width for more reliable detection of the tote edge. It draws a thicker outline around the detected tote and helps filter out noise from the narrow border detection.

Edge filter radius (Advanced)

The radius (in pixels) used to decide whether a point is on the edge of the tote. The algorithm checks the points along a line towards or away from the centroid (depending on Tote edge detection point set). If all pixels within this radius are empty, the point is an edge point. Larger values make the filter stricter: more empty space is needed to detect an edge.

Tote rim plane inliers distance tolerance (Advanced)

The largest distance a 3D point can be from the detected tote rim plane and still count as part of it. Lower values make the plane detection stricter. Higher values are more tolerant, but also accept more outliers.

Minimum plane inliers threshold (Advanced)

The lowest percentage of points of the segmented tote edge that must lie on the detected plane for the plane to be valid. This makes sure the plane really represents the top of the tote.

Expected plane inliers threshold (Advanced)

The percentage of points of the segmented tote edge that you expect to lie on the detected plane. This estimate helps the plane search run faster.

Plane sampled points count (Advanced)

The number of points picked at random from the segmented tote edge in each step of the plane search, to estimate a plane on top of the tote. More points can give a more accurate result, but take longer to compute.

General methods

These settings apply to Centroid, Largest Inscribed Circle and Keypoint from NN.

Normal radius (Basic)

The normal (Z-direction) of the gripping point is the average of the normals of the 3D points within this radius.

Surface roughness compensation multiplier (Basic)

Moves the gripping point towards the object surface, along the Z-axis of the gripping point. The distance is this value times the surface roughness (the standard deviation of the depth). This can help when gripping soft objects with uneven surfaces.

Force the picking direction vector (Basic)

When enabled, the Z-axis of the gripping point is forced to:

• the Gripper reference vector (AI Picking solution type)

• the opposite of the Angle sorter vector (AI Localization solution type)

When the orientation is forced, the sorter weight has no effect, so you can set it to 0.

Suction cup radius (Advanced)

The radius of the suction cup (the gripper). Used only when Coverage weight is not zero.

Coverage weight (Advanced)

How important it is that the suction cup (of the Suction cup radius) is well covered by the surface. The higher the value, the more the engine prefers gripping points on a surface that covers the suction cup well.

Distance weight (Advanced)

How important it is that the gripping point stays close to the initial gripping point. The higher the value, the closer to the initial gripping point the engine places it.

Surface roughness weight (Advanced)

How important a smooth surface is. The higher the value, the more the engine prefers smoother surfaces for the gripping point.

Dimensions reporting

The engine can report the X and Y dimensions of the smallest bounding box around each found object. For rectangular objects, such as boxes or totes, these are close to the real dimensions of the object.

With the dimensions, the engine also reports the angle between the positive X-axis of the object and the positive X-axis of the reference frame (the calibration frame). The angle is 0–360 degrees and increases from the X-axis towards the Y-axis of the reference frame.

Four situations: the axes of the reference frame (top), the axes of the found object (in the middle of the box) and the computed angle in degrees in the Inspector widget (orange)